Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Normalize HTTP lifecycle handlers with HTTP probers #86139

Conversation

jasimmons
Copy link
Contributor

@jasimmons jasimmons commented Dec 10, 2019

What type of PR is this?
/kind bug

What this PR does / why we need it:

Align the behavior of HTTP-based lifecycle handlers and HTTP-based probers, converging on the probers implementation. This fixes multiple deficiencies in the current implementation of lifecycle handlers surrounding what functionality is available.

The functionality is gated by the features.ConsistentHTTPGetHandlers feature gate.

Which issue(s) this PR fixes:

Similar to #68846

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Action required: Container `preStop` and `postStart` lifecycle handlers using `httpGet` now honor the specified `scheme` and `headers` fields. This enables setting custom headers and changing the scheme to `HTTPS`, consistent with container startup/readiness/liveness probe capabilities. Lifecycle handlers configured with `scheme: HTTPS` that encounter errors indicating the endpoint is actually using HTTP fall back to making the request over HTTP for compatibility with previous releases. When this happens, a `LifecycleHTTPFallback` event is recorded in the namespace of the pod and a `kubelet_lifecycle_handler_http_fallbacks_total` metric in the kubelet is incremented. Cluster administrators can opt out of the expanded lifecycle handler capabilities by setting `--feature-gates=ConsistentHTTPGetHandlers=false` in `kubelet`.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 10, 2019
@k8s-ci-robot
Copy link
Contributor

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Dec 10, 2019
@k8s-ci-robot
Copy link
Contributor

Welcome @jasimmons!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot
Copy link
Contributor

Hi @jasimmons. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Dec 10, 2019
@jasimmons jasimmons changed the title Pr normalize probes lifecycle handlers Normalize HTTP lifecycle handlers with HTTP probers Dec 10, 2019
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Dec 10, 2019
@jasimmons jasimmons force-pushed the pr_normalize_probes_lifecycle_handlers branch from 9b3088f to 13eca1a Compare December 10, 2019 23:21
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. label Dec 10, 2019
Copy link
Contributor

@mattjmcnaughton mattjmcnaughton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jasimmons !

thanks for your pr and welcome to kubernetes :)

Before reviewing the code in too much depth, I want to be sure I'm understanding the purpose of this pr :)

Can you share a little more about the specific bug(s) that we're trying to solve? Is it that we are not respecting headers passed to HTTPGetAction.

Additionally, when you describe this diff as "normalizing", can you share more about what we are normalizing w.r.t? My initial thought was that other parts of k8s are using HttpDoer, so we would be unifying behavior in this respect. However, I don't see any existing mentions of HttpDoer.

/ok-to-test

pkg/kubelet/lifecycle/handlers.go Outdated Show resolved Hide resolved
pkg/kubelet/lifecycle/handlers.go Outdated Show resolved Hide resolved
pkg/kubelet/types/types.go Outdated Show resolved Hide resolved
@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 11, 2019
@jasimmons
Copy link
Contributor Author

Hey, @mattjmcnaughton , thanks! :)

Can you share a little more about the specific bug(s) that we're trying to solve? Is it that we are not respecting headers passed to HTTPGetAction.

That's part of it, yes. Also, though, lifecycle handlers always prepend a leading / in their path, which is different than the behavior of probers.

As an example, here's a pod:

apiVersion: v1
kind: Pod
metadata:
  name: sample-webserver
spec:
  containers:
    - name: webserver
      image: alpine:3
      command: ["/bin/sh"]
      args: ["-c", "while true; do echo 'HTTP/1.1 204 No Content' | nc -l -p 80; done"]
      ports:
        - name: tcp-80
          containerPort: 80
          protocol: TCP
      readinessProbe:
        httpGet:
          path: /ready
          port: 80
          scheme: HTTP
          httpHeaders:
            - name: foo
              value: bar
      lifecycle:
        preStop:
          httpGet:
            path: /preStop
            port: 80
            scheme: HTTP
            httpHeaders:
              - name: foo
                value: bar

When applying this and viewing the logs of this pod, you can see the path is as-expected and HTTP headers are passed appropriately for the readinessProbe:

GET /ready HTTP/1.1
Host: 10.38.36.93:80
User-Agent: kube-probe/1.16
foo: bar
Accept-Encoding: gzip
Connection: close

However, when terminating this pod, the log shows that the path always has a / prepended, and the HTTP headers are not passed in the preStop hook:

GET //preStop HTTP/1.1
Host: 10.38.36.93:80
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

Additionally, when you describe this diff as "normalizing", can you share more about what we are normalizing w.r.t? My initial thought was that other parts of k8s are using HttpDoer, so we would be unifying behavior in this respect. However, I don't see any existing mentions of HttpDoer.

The normalization here is the behavior of that of httpGet actions for probers and lifecycle handlers. My assumption is that probers do what is expected, but lifecycle handlers do not. "What is expected" in this context is the proper formatting of the URL (prepending a / if necessary, but not if not) and passing HTTP headers in the request.

The HttpDoer interface is replacing an HttpGetter to provide the same functionality but with the ability to accept HTTP headers. HttpGetter similarly was not used anywhere outside of this, so I'm certainly open to trying to find a new home for the interface.

Thanks for the feedback!

@jasimmons jasimmons force-pushed the pr_normalize_probes_lifecycle_handlers branch 2 times, most recently from a426502 to 3bcf884 Compare December 11, 2019 16:26
Copy link
Contributor

@mattjmcnaughton mattjmcnaughton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha gotcha, ok that additional information is helpful - thank you :)

In my mind, there's a clear benefit to fixing the issues you outlined :) So its just a matter of finding the best way to do so.

One high level question I have - did we consider whether it would be possible to extract some functionality from pkg/probe/http/http.go into a shared location, and then have both the lifecycle handlers and the http probers rely on it? Specifically some of the func here: https://github.com/kubernetes/kubernetes/blob/master/pkg/probe/http/http.go#L82.

I think part of deciding whether extracting common functionality would be wise is understanding if we're concerned about further drift between HTTP lifecycle handlers and HTTP probers in the future. Do you have thoughts on that topic?

Also, very curious to hear from others here :)

@liggitt
Copy link
Member

liggitt commented Oct 19, 2022

change lgtm, go ahead and squash down to the commits you want for merge and open a tracking issue for the two follow-ups (for the metric and the event)

@logicalhan
Copy link
Member

change lgtm, go ahead and squash down to the commits you want for merge and open a tracking issue for the two follow-ups (for the metric and the event)

Happy to look over the metric once you have something ready.

jasimmons and others added 2 commits October 19, 2022 09:51
Align the behavior of HTTP-based lifecycle handlers and HTTP-based
probers, converging on the probers implementation. This fixes multiple
deficiencies in the current implementation of lifecycle handlers
surrounding what functionality is available.

The functionality is gated by the features.ConsistentHTTPGetHandlers feature gate.
@bhcleek
Copy link
Contributor

bhcleek commented Oct 19, 2022

I've squashed the changes down to two commits. No other changes were made since the last review.

The follow up issue is #113174.

@liggitt
Copy link
Member

liggitt commented Oct 19, 2022

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 19, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, jasimmons, liggitt, ohbus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 19, 2022
@liggitt
Copy link
Member

liggitt commented Oct 19, 2022

/retest

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Oct 19, 2022

@jasimmons: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kubernetes-cross 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-cross
pull-kubernetes-local-e2e 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-local-e2e
pull-kubernetes-e2e-aks-engine-azure 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-aks-engine-azure
pull-kubernetes-e2e-gce-storage-slow 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-gce-storage-slow
pull-kubernetes-e2e-gce-csi-serial 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-gce-csi-serial
pull-kubernetes-e2e-gce-iscsi 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-gce-iscsi
pull-kubernetes-e2e-gce-iscsi-serial 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-gce-iscsi-serial
pull-kubernetes-e2e-azure-disk-vmss 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-azure-disk-vmss
pull-kubernetes-e2e-azure-file 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-azure-file
pull-kubernetes-e2e-gce-storage-snapshot 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-gce-storage-snapshot
pull-kubernetes-e2e-azure-file-windows 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-azure-file-windows
pull-kubernetes-e2e-azure-disk-windows 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-azure-disk-windows
pull-kubernetes-e2e-gci-gce-autoscaling 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-gci-gce-autoscaling
pull-publishing-bot-validate 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-publishing-bot-validate
pull-kubernetes-e2e-azure-disk 17740200044b0d9cdb098aff85dc5ea69f2836cb link /test pull-kubernetes-e2e-azure-disk
pull-kubernetes-node-kubelet-serial-containerd 180fc262508aa9eb47afefeb0bd56b4681d52b52 link /test pull-kubernetes-node-kubelet-serial-containerd
pull-kubernetes-node-kubelet-serial 180fc262508aa9eb47afefeb0bd56b4681d52b52 link /test pull-kubernetes-node-kubelet-serial

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@bhcleek
Copy link
Contributor

bhcleek commented Oct 19, 2022

/test pull-kubernetes-e2e-gce-ubuntu-containerd

@k8s-ci-robot k8s-ci-robot merged commit ad26b31 into kubernetes:master Oct 20, 2022
SIG Node CI/Test Board automation moved this from Archive-it to Done Oct 20, 2022
SIG Node PR Triage automation moved this from Needs Approver to Done Oct 20, 2022
SIG Auth Old automation moved this from Changes Requested to Closed / Done Oct 20, 2022
@liggitt
Copy link
Member

liggitt commented Oct 20, 2022

Updated release note with event and metric name added in #113175

ionutbalutoiu added a commit to ionutbalutoiu/kubernetes that referenced this pull request Nov 3, 2022
In the PR kubernetes#86139, two more lifecycle hook tests (poststart / prestop)
were added using HTTPS. They are similar with the existing HTTP tests.

However, this causes failures on Windows due to how networking
works there. We previously fixed this in the HTTP tests via kubernetes@f9e4a01.

This commit applies the same fix on the lifecycle hook HTTPS tests.

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>
@k8s-ci-robot k8s-ci-robot added release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Dec 6, 2022
jaehnri pushed a commit to jaehnri/kubernetes that referenced this pull request Jan 3, 2023
In the PR kubernetes#86139, two more lifecycle hook tests (poststart / prestop)
were added using HTTPS. They are similar with the existing HTTP tests.

However, this causes failures on Windows due to how networking
works there. We previously fixed this in the HTTP tests via kubernetes@f9e4a01.

This commit applies the same fix on the lifecycle hook HTTPS tests.

Signed-off-by: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver area/cloudprovider area/conformance Issues or PRs related to kubernetes conformance tests area/dependency Issues or PRs related to dependency changes area/e2e-test-framework Issues or PRs related to refactoring the kubernetes e2e test framework area/ipvs area/kubectl area/kubelet area/provider/gcp Issues or PRs related to gcp provider area/release-eng Issues or PRs related to the Release Engineering subproject area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. sig/cli Categorizes an issue or PR as relevant to SIG CLI. sig/contributor-experience Categorizes an issue or PR as relevant to SIG Contributor Experience. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/network Categorizes an issue or PR as relevant to SIG Network. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/release Categorizes an issue or PR as relevant to SIG Release. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
Archived in project
Archived in project
Archived in project
SIG Auth Old
Closed / Done
Development

Successfully merging this pull request may close these issues.

None yet